Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[tempo-distributed] fix: add autoscaling for tempo-distributed metrics-generator #3430

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

msvechla
Copy link

This adds autoscaling via hpa and keda for tempo-distributed metrics-generator. Implementation is analog to the already existing autoscaling options for e.g. the compactor.

HPA Example

helm template tempo . --set metricsGenerator.enabled=true --set metricsGenerator.autoscaling.enabled=true --set metricsGenerator.autoscaling.hpa.enabled=true --show-only templates/metrics-generator/hpa.yaml
---
# Source: tempo-distributed/templates/metrics-generator/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: tempo-metrics-generator
  namespace: default
  labels:
    helm.sh/chart: tempo-distributed-1.22.0
    app.kubernetes.io/name: tempo
    app.kubernetes.io/instance: tempo
    app.kubernetes.io/component: metrics-generator
    app.kubernetes.io/version: "2.6.0"
    app.kubernetes.io/managed-by: Helm
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: tempo-metrics-generator
  minReplicas: 1
  maxReplicas: 3
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 100

KEDA Example

helm template tempo . --set metricsGenerator.enabled=true --set metricsGenerator.autoscaling.enabled=true <...> --show-only templates/metrics-generator/keda-scaled-object.yaml
# Source: tempo-distributed/templates/metrics-generator/keda-scaled-object.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: tempo-metrics-generator
  namespace: default
  labels:
    helm.sh/chart: tempo-distributed-1.22.0
    app.kubernetes.io/name: tempo
    app.kubernetes.io/instance: tempo
    app.kubernetes.io/component: metrics-generator
    app.kubernetes.io/version: "2.6.0"
    app.kubernetes.io/managed-by: Helm
spec:
  minReplicaCount: 1
  maxReplicaCount: 3
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: tempo-metrics-generator
  triggers:
  - type: "prometheus"
    metadata:
      serverAddress: http://<prometheus-host>:9090
      threshold: "250"
      query: |
        sum(prometheus_remote_storage_shards_desired{job="default/metrics-generator"} /
        prometheus_remote_storage_shards_max{job="default/metrics-generator"})by(job)

Let me know if this can be merged or if further adjustment is required.
Thanks!

This adds autoscaling via hpa and keda for tempo-distributed
metrics-generator. Implementation is analog to the already existing
autoscaling options for the compactor.

Signed-off-by: Marius Svechla <[email protected]>
@Sheikh-Abubaker Sheikh-Abubaker changed the title fix: add autoscaling for tempo-distributed metrics-generator [tempo-distributed] fix: add autoscaling for tempo-distributed metrics-generator Nov 15, 2024
@msvechla
Copy link
Author

I rebased

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant